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Abstract 

Big networks express multiple classes of large-scale networks in many practical ar¬ 
eas such as computer networks, internet of things, cloud computation, manufacturing 
systems, transportation networks, and healthcare systems. This paper analyzes such 
big networks, and applies the mean-field theory and the nonlinear Markov processes 
to constructing a broad class of nonlinear continuous-time block-structured Markov 
processes, which can be used to deal with many practical stochastic systems. Firstly, a 
nonlinear Markov process is derived from a large number of big networks with weak in¬ 
teractions, where each big network is described as a continuous-time block-structured 
Markov process. Secondly, some effective algorithms are given for computing the fixed 
points of the nonlinear Markov process by means of the UL-type i?G-factorization. 
Finally, the Birkhoff center, the locally stable fixed points, the Lyapunov functions 
and the relative entropy are developed to analyze stability or metastability of the 
system of weakly interacting big networks, and several interesting open problems are 
proposed with detailed interpretation. We believe that the methodology and results 
given in this paper can be useful and effective in the study of big networks. 

Keywords: Nonlinear Markov process; Big network; Mean-field theory; i?G-factorization; 
Fixed point; Stability; Metastability; Lyapunov function; Relative entropy 


1 Introduction 

In this paper, we consider a large number of big networks with weak interactions, where 
each big network is described as a continuous-time block-structured Markov process, which 
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can be applied to deal with many practical stochastic systems. As the number of big net¬ 
works goes to infinite, the interactions between any two subsets of the big networks become 
negligible or are asymptotically independent, and the overall effect of the interactions can 
be replaced by an empirical measure under a mean-field setting. Based on this, the evolu¬ 
tion of such a big network is expressed as a time-inhomogeneous block-structured Markov 
process, which leads to that transient performance of the big network can be discussed 
by means of a system of ordinary differential equations, while its stationary performance 
measures can be computed in terms of a fixed point, which satisfies a system of nonlinear 
equations. 

The purpose of this paper is to develop the mean-field computational theory both 
for performance evaluation and for performance optimization of big networks. During 
the last three decades considerable attention has been paid to studying the mean-field 
theory of large-scale stochastic systems, which has been well documented, for example, 
an excellent survey paper by Sznitman [ZH]; interacting Markov processes by, such as, 
Spitzer [76], Kurtz [52l|53l[54], Dawson [23], Shiga and Tanaka [75], Dawson and Zheng 
[24] . Duffield and Werner [29], Duffield [28], Le Boudecet at al. [55], Darling and Norris 
[22], Bordenave at al. m, Benaim and Le Boudec mm and Li [57]; interacting Markov 
decision processes by Cast and Gaujal [38] and Cast at al. [39]; and more generally, 
interacting particle systems introduced by three books of Kipnis and Landim [48] , Chen 
m and Liggett [62]. For limit theory of stochastic process sequences, readers may refer 
to three books of Ethier and Kurtz m, Whitt [85], and Jacod and Shiryaev [45] . 

During the last two decades the mean-field theory has widely been applied to studying 
some practical networks (or systems) including queueing systems, communication net¬ 
works, manufacturing systems, transportation networks and so forth, (a) For queueing 
systems, Baccelli et al. [1] and Kelly m first applied the mean-field theory to the study 
of queueing networks. Subsequent papers have been published on this theme, among 
which see Borovkov m, Bobbio et al. m-, Delcoigne and Fayolle [25j for polling sys¬ 
tems; Karpelevich and Rybko [l6] for symmetric closed queuing networks; Baccelli et al. 
[5] for varying topology networks; Hayden et al. [33] for stochastic process algebra; and 
Hunt and Kurtz [33], and Zachary and Ziedins [86] for large loss networks, (b) To discuss 
randomized load balancing, some work has been done on two different research directions: 
Supermarket models, and work stealing models. For the supermarket models, readers may 
refer to, such as, Vvedenskaya et al. [83] for operator semigroup, Mitzenmacher [68] for 
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density-dependent jump Markov processes, Turner [80] for martingale limits. Subsequent 
papers have been published on this theme, important examples include Vvedenskaya and 
Suhov [8l|, Graham [HI |32|, Luczak and McDiarmid [Ml IM]) Li et al. [58l EH] and Li 
|60j . As a key generalization of the supermarket models, the fast Jackson networks were 
investigated by Martin and Suhov [M]) Martin [Mj, and Suhov and Vvedenskaya [77] • In 
contrast, the available results of the work stealing models based on the mean-field theory, 
Markov processes and queueing theory are still few in the literature, e.g., see Cast and 
Gaujal m and Li and Yang m- In addition, readers may refer to the computer and com¬ 
munication systems by Benaim and Le Boudec OHO], and Antunes et al. mm-, the bike 
sharing systems by Fricker et al. [35| and Fricker and Gast [3l|; and the transportation 
networks by Oseledets and Khmelev m- 

Nonlinear Markov processes play an important role in the study of big networks. Im¬ 
portant examples include Rybko and Shlosman m, Peng m, Turner [81]; Benaim and 
Le Boudec [9], Frank [32] j Kolokoltsov [39], Gast and Gaujal [38]) Kolokoltsov [IS [50], 
Kolokoltsov at al. m, Muzychka and Vaninsky [69j . Dupuis and Fischer [30], Gast at 
al. [39], Vaninsky et al. [82], Budhiraja et al. [IS [19], Budhiraja and Majumder m and 
Benaim [8]. 

Metastability is an ubiquitous and important phenomenon of the dynamical behavior 
of communication networks, e.g., see Marbukh [67j . Gibbens at al. |40j . Kelly m (see 
Page 349), Dobrushin [27]) Antunes et al. [3] and Tibi |79j . For metastability in Markov 
processes, readers may refer to Galves at al. [36], Bovier et al. [Ml ET], Olivieri and 
Vares [M]) Freidlin and Wentzell [33] ) Bovier [MlEl]) den Hollander |26j , and Beltran and 
Landim PE]. 

The main contributions of this paper are threefold. The first one is to set up a broad 
class of nonlinear continuous-time block-structured Markov processes when applying the 
mean-field theory to analysis of big networks with weak interactions. The second one is to 
propose some effective algorithms for computing the hxed points of the nonlinear Markov 
processes by means of the UL-type i?G-factorization, and show for some big networks that 
there possibly exist multiple hxed points, which lead to the metastability. The third one is 
to use the Birkhoff center, the locally stable hxed points, the Lyapunov functions and the 
relative entropy to analyze either stability or metastability of the big networks, and to give 
several interesting open problems with detailed interpretation. Furthermore, this paper 
provides a new method for computing the locally stable hxed points in the study of big 
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networks. We believe that the methodology and results given in this paper can be useful 
and effective in performance evaluation and performance optimization of big networks. 

The remainder of this paper is organized as follows. In Section 2, we derive a class of 
nonlinear Markov processes through an asymptotic analysis of the weakly interacting big 
networks. In Section 3, we provide some effective algorithms for computing the fixed points 
of the dynamic system of mean-field equations. In Section 4, we discuss the Birkhoff center 
and the locally stable fixed points of the dynamic system of mean-field equations, and apply 
the Lyapunov functions and the relative entropy to study the stability or metastability 
of the big network. Also, we provide several interesting open problems with detailed 
interpretation. Some concluding remarks are given in the final section. 


2 Nonlinear Markov Processes 


In this section, we derive a class of nonlinear Markov processes through an asymptotic 
analysis for a collection of weakly interacting big networks, in which each big network 
evolves as a continuous-time block-structured Markov process, which can be applied to 
deal with many practical stochastic systems. 

To discuss a system of weakly interacting big networks, we assume that any individual 
of the big networks evolves as a continuous-time block-structured Markov process A whose 
infinitesimal generator is given by 


Q = 


Qo,o 

Qo,i 

Qo,2 

Qo,3 ■ ■ ■ 


Qi,o 

Qi,i 

Qi,2 

Qi,3 ■ ■ ■ 


Q2,0 

Q2,1 

Q2,2 

Q2,3 ■ ■ ■ 

(1) 

Q3,0 

Qs,! 

Q3,2 

Q3,3 ■ ■ ■ 





■ / 


i,j is 

nij for 

j > 

0, and the sizes of other matrices can 


be determined accordingly. It is easy to see that the matrix Qjj is also the infinites¬ 
imal generator of a continuous-time Markov process with rrij states for j > 0. We 
assume that the continuous-time Markov process Q is irreducible, aperiodic and pos¬ 
itive recurrent, and its state space may be expressed as a two-dimensional structure: 
^ = {{k,j) '■ k > 0,1 < j < rrik}- See Li [56] for more details. 

From the continuous-time block-structured Markov chain X, the system of N weakly 
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interacting big networks is described as an '^-valued Markov process, and the states of 
the N big networks are denoted as (t), (t), .. X^’^ (t), respectively. 

Let (t) = (t) , X'^’^ (t),..., X^’^ (t)). Then the empirical measure of the 

system of N weakly interacting big networks is given by 

1 ^ 

(^) = (t )) ( 2 ) 

i=l 

where 5x is the Dirac measure at x. 

We denote by ^ (D) the space of probability vectors on the state space D, which 
is equipped with the usual topology of weak convergence. If p G we write p = 

{Po,PiiP 2 , ■ ■ ■), where the size of the vector pj is rrij for j > 0. At the same time, it 
is clear that (t) G ^ (D) is a random variable for t > 0, and (t) '■ t > O} is a 
continuous-time Markov process. 

For the A^-valued continuous-time block-structured Markov process, we define that 
the probability distribution of (t) is exchangeable, if for any level permutation {kp , fcjj,..., 
ki^) of {ki,k 2 , ..., kw) and any phase permutation {jpjp,. ■ ■ of (ji, j 2 , • • • Jn), 

P (t) = (fci, ji) ,X2'^ (t) = {k2,j2) , . . it) = {kN,jN)} (3) 

= P [X^^X (i) = (i) = ^^ . 

In the system of N weakly interacting big networks, the effect of a tagged big network 
on the dynamics of the system of N weakly interacting big networks is of order 1/X, and 
the jump intensity of any given big network depends on the configuration of other big 
networks only through the empirical measure p^ (t). To study the system of N weakly 
interacting big networks, it is seen from probability one that at most one big network 
will jump, i.e., change state, at a given time, and the jump intensities of any given big 
network depend only on its own state and the state of the empirical measure at that 
time. In addition, the jump intensities of the N weakly interacting big networks have 
the same functional form. Based on this, for the X'^-valued Markov process, if the initial 
probability distribution of X^ (0) is exchangeable, then at any time t > 0, the probability 
distribution of X^ (t) is also exchangeable. 

For the system of N weakly interacting big networks, if the probability distribution 
of X^ (t) is exchangeable, then the N big networks are indistinguishable, thus we apply 
the mean-field theory to discussion of this system through only considering the Markov 


5 



process of a tagged big network (such as, the first big network); while analysis of the total 
system can be completed by the propagation of chaos (as N —>■ oo). Based on this, the 
inhnitesimal generator of the Markov process corresponding to the tagged big network is 
defined as 


(t)) 


C ((■" «)) 
rff (a-" (*)) 
by (a-" (t)) 

by ((•"«)) 


byb^w) 

rsy 

by b"(‘)) 

by b'^w) 


by (m" («)) 

rsy (f.-^ (t)) 

by (a-" (‘)) 
by («)) 


by («"«)) 

r!y (k" (t)) 

by (f*" (‘)) 
by {(<" (*)) 


... t 


/ 

(4) 


where the size of the matrix (/i^ (t)) is ruj for j > 0, and the sizes of other ma¬ 
trices can be determined accordingly. Since (t) is a random variable, it is clear that 
rw (t)) is a random matrix of infinite order. On the other hand, it is seen from the 
law of large number that the limit of the empirical measure (t) is deterministic under 
suitable conditions. 

Now, we analyze some convergence of the sequence (t) : t > 0} of Markov pro¬ 
cesses for N" = 1, 2, 3,..., and our aim is to provide a basic support for our later study of 
various convergence involved. To this end, we consider the empirical measure: (t) = 

W with samples in V (B {R+, N)), where R+ = [0, -boo), N = {{k, j) : k > 0,1 

and B(i2+,N) is the Skorohod space, i.e., the set of mappings x : R+ —>■ N which are 
right continuous with left-hand limits (in short, cadlag). Readers may refer to Chapter 3 
of Ethier and Kurtz m for more details. Notice that the convergence in the Skorohod 
topology means the convergence in distribution (or weak convergence) for the Skorohod 
topology on the space of trajectories, we assume that the sequence (t) : t > O} of 
Markov processes converges in probability (or converges weakly), for the Skorohod topol¬ 
ogy, to a given probability vector p{t). At the same time, for this weak convergence, we 
write (t) =b p (t) for t > 0, as oo. 

Let (t) =b p (t) and (t)) =b T {p (t)) for t > 0, as N ^ oo. Then p (t) is 

a given probability vector. Furthermore, using some probability analysis, we may obtain 
an infinite-dimensional dynamic system of mean-field equations as follows: 


^p{t) =p{t)r{p{t)) 


(5) 


<j< ruk}, 
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with the initial condition 


p(0) = q. 


( 6 ) 


Obviously, the dynamic system of mean-field equations, given in ([5]) and ([6]), is related to 
a nonlinear Markov process whose infinitesimal generator is given by 


r{p{t)) 


^ '^o,o{p{t)) To,i{p{t)) ro^2 {p{t)) ro,3(p(i)) 

ri,o(p(i)) ri,i(p(t)) ri^2 {p{t)) Ti^3{p{t)) 

'^2,0 {p{t)) T2,l{p{t)) T2,2{p{t)) T2^3{p{t)) 

^3,0 (pit)) ^3,1 (pit)) ^3,2ip{t)) r3,3(p(t)) 


V 


... \ 


/ 


(7) 


Remark 1 To establish the infinitesimal generator T {p{t)) of a nonlinear Markov pro¬ 
cess, readers may also refer to some recent publications, for example, the discrete-time 
Markov chains by Benaim and Le Boudec m and Budhiraja and Majumder JWf . the 
Markov decision processes by Cast and Gaujal ffiBi and Cast at al. the continuous¬ 
time Markov chains by Dupuis and Fischer JSUf and Budhiraja et al. and some 

practical examples include Mitzenmacher JEEf, Bobbio et al. m, Li et al. [5^, and 
Li and Lui J6(^ . 


In what follows, it is necessary to provide some useful interpretation or proofs for how 
to establish the dynamic system of mean-field equations ([5]) and (|6|). 

(a) Existence and Uniqueness 

Consider the infinite-dimensional ordinary differential equation: ^p (t) = p{t)T {p (t)) 
with p (0) = q. A solution in the classical sense is a (continuously) differential function 
p{t) such that ^p{t) = p{t)T{p{t)) with p{0) = q. A classical method is the Picard 
approximation as follows. If P (x) is (locally) Lipschitz on a set E C ^ (II), that is, there 
exists a positive constant C such that 

IIP (x) - r (?/)|| < Cllx - y|| , x,7/ G (H) , 

and p (0) = q is in the interior of E, then there exists a unique global solution to the 
ordinary differential equation: ^p (t) = p{t)T (p (t)) with p (0) = q, within E. 

To deduce whether the P (x) is (locally) Lipschitz on a set E C ip (II), Li et al. [58] 
and Li and Lui |60j gave an algorithmic method through dealing with some matrices of 
infinite orders. 
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(b) The limiting processes 

To discuss the limit: For t > 0, (t) => p{t) as N —>■ oo, we need to set up some 

suitable conditions in order to guarantee the existence of such a limit. 

Let Cfej be the unit vector of infinite dimension in which the {k,j)ih. entry is one and 
all the others are zero. Note that the empirical measure process (t) : t > O} 

is a Markov process on the state space (Q) where = ip (O) n the 

possible jumps of are of the form — ekj) /N for {l,i) / {k,j), and {k,j) , {l,i) G 
n. If {t) = X G iPw (f^) at time t > 0, then Nxkj denotes State {k,j) of the big 
network. Hence the transition rate of the Markov process from State {k,j) to State 
corresponding to the tagged big network, is given by ■ (x). Based on this, the 

generator of the Markov process is given by 


A^^^f{x)= 






where / (x) is a real function on ipw (fl), and there are two types of boundary states: That 
a task enters the big network corresponds to an arriving boundary state: {k,j) = (0,j); 
while a task is completed and immediately leaves the big network corresponds to a departed 
boundary state: {l,i) = (0,i). It is easy to see that as N ^ oo 


aW/(^ 


^k,j^k,j-,l,i (x) 


{k,j)GU {l,i)£Q 


d 


dxi^i 


fix) - 


d 


dx 


k,j 


-fix) 


def 


= A/(x). 


Theorem 1 Suppose that for {k,j),{l,i) G Cl with ik,j) ^ il,i), there exists a Lips- 
chitz continuous function Fkj-i^iip) : ^ (f^) ^ [0,+oo) such that —)> Tkj-i^iip) 

uniformly on fp (Cl). If (0)} converges in probability to G ip iCl), then (t)} 

converges uniformly on compact time intervals in probability to p{t) G ip (Cl) for f > 0, 
where the probability vector p (t) is the unique global solution to the ordinary differential 
equation: -^p (t) = p{t)T {p (t)) with p (0) = q. 


Proof: The proof may directly follow from Theorem 2.11 in Kurtz 
give a simple outline as follows. Firstly, notice that 


. Here, we only 


and 


ip) = ip) 

Fip) = ^ Pkj iei,i - ek,j) iP) > 

(k,j),{l,i)&n 
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where as —>■ oo 


and r (x) is (locally) Lipschitz on a set £1 C ip (Q), thus for the sequence > O} 

of Markov processes, it follows from Equation (III. 10.13) in Rogers and Williams |73] or 
Page 162 in Ethier and Kurtz [31] that 

(t) = ^(^)(x)r dx 

is a martingale with respect to each > 1. Therefore, if |^*'^)(0)} converges weakly 
to O' € fp (n) as Ai —>■ oo, then |^(^)(t), A^ > 1} converges weakly in +oo) endowed 

with the Skorohod topology to the solution p (t) to the ordinary differential equation: 
^P i't) = p{t)^ (p (t)) with p (0) = q, within ip (Q). This completes the proof. ■ 

Remark 2 Benaim and Le Boudec applied the mean-field theory to studying the discrete¬ 
time system of N weakly interacting objects, and for t = 0,1,2,the states of this en¬ 
tire system are expressed as (t) = (t), (t),..., X^"' (t ); RX) (t)^ where 

(t) is the state of the kth object for 1 < k < N, and rX) (t) is the state of the 
common resource of the N objects. Differently from that of Benaim and Le Boudec m 
this paper uses the mean-field theory to studying the continuous-time system of N weakly 
interacting objects, where the states of each object is described as a block-structured Markov 
process (i.e., a Markov process under a stochastic enviornment jX) and it is easy 
to see that the stochastic enviornment jX) (t) may be regarded as the resource R^^'l (t). 
On the other hand, we provide a simple method to describe the weak interaction of the 
N objects through introduction to the matrix T^-^^ (p-^ (t)) where (t) is the empirical 
measure of the system of the N weakly interacting objects. Although our method is simple 
to deal with the weak convergence T^-^^ {t)) T {p{t)) as N ^ oo, it is useful and 

effective in the mean-field study of many continuous-time big networks, readers may refer 
to Li et al. imw and Li fWf for more details. 

3 The Fixed Points 

In this section, we use the UL-type RG-factorization to provide some effective algorithms 
for computing the fixed points of the ordinary differential equation: ^p (t) = p (t) T [p (t)) 
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with p (0) = q. Further, we set up a nonlinear characteristic equation of the censoring 
matrix to level 0, which is satisfied by the fixed points. 

A point TT € ^ (n) is said to be a fixed point of the ordinary differential equation: 
{t) = P (i) r {p (t)) with p (0) = g, if p (t) —)■ TT as t ^ +oo, and 


lim 

t —^H~oo 



= 0 . 


In this case, it is clear that 


ttF (tt) = 0, 


( 8 ) 


which is an infinite-dimensional system of nonlinear equations. In general, there are more 
difficulties and challenging due to both the infinite order of and the nonlinear structure of 
the matrix F (tt) when solving the fixed point equation ([8|) together with vre = 1, where e 
is a column vector of ones with a suitable size. 

It is easy to check that for every vr € ip (H), F (tt) is the infinitesimal generator of an 
irreducible continuous-time Markov process. Based on Li [56], we can develop the UL-type 
i?G-factorization of the matrix F (vr). To that end, we partition the matrix F (tt) as 

y V{tt) W{tt) j 

according to the level sets and L>„_|_i for n > 0. Since the Markov chain F (tt) is 
irreducible, it is clear that the two truncated chains with infinitesimal generators T (tt) 
and W (tt) are all transient, and also the matrices T (vr) and W (vr) are all invertible from 
a different understanding that the inverse of the matrix T (vr) is ordinary, but the invert- 
ibility of the matrix W (vr) is different under an infinite-dimensional meaning. Although 
the matrix W (vr) of infinite size may have multiple inverses, we in general are interested 
in the maximal non-positive inverse W~l^ (vr) of W (vr), i.e., W~^ (vr) < W~l^ (vr) < 0 for 
any non-positive inverse W~^ (vr). Of course, 0 < [—IT (vr)]~[jj < [—IT (vr)]~^ for any non¬ 
negative inverse [—IT (vr)]~^ of —IT (vr), that is, [—IT (vr)]”^^^ is the minimal nonnegative 
inverse of —IT (vr). Based on this, for n > 0 we write 

(^) • • • 45 (^) ^ 

(tt) • • • (/>S5 (^) 

. . 1 

••• 45 (^) ) 


r[^"l (vr) = T (vr) + [/ (vr) [-IT (vr)]"!, T (vr) = 


45 (^) 
45 (^) 


45 (^) 
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where the size of the matrix (vr) is rrij for 0 < j < n, and the sizes of other matrices 
can be determined accordingly. It is clear from Section 7 of Chapter 2 in Li |56j that for 
n > 0 , 0 < i, j < n, 



OO 

(vr) = Lij (vr) + ^ (vr) 

k=n-\-l 




Let 


(tt) = (vr), n > 0 ; 


Rij (vr) = M -4>t] (tt) 


/.(C 


1 -1 




and 


Gi,j (vr) = - 4 ? (vr)] 4>^] (vr), 0 < j < i. 


Then the UL-type i?G-factorization of the matrix L (vr) is given by 


r (tt) = [I-Ru (vr)] (vr) [/ - Gl (vr)] , 


(9) 


where 


and 


Ru (vr) = 


^ 0 Ro^i (vr) i?o ,2 (vr) Ro,3 (vr) • • • ^ 

0 Ri ^2 (vr) Ri ,3 (vr) • • • 

0 R2,3 (vr) • • • 

0 


V 


Tz) (vr) = diag (Tq (tt) , Ti (vr), T 2 (vr), T 3 (tt) 


Gl (vr) = 


/ 0 

Gpo (vr) 0 
1 ^ 2,0 (vr) ^ 2,1 (vr) 0 
(^ 3,0 (vr) Gsp (vr) ^ 3,2 (vr) 0 

V • • • • 






Based on the UL-type i?G-factorization ([9]), it follows from Subsection 2.7.3 in Li [56 
that the fixed point vr is given by 


vro = txq (vr), 

< k-i ( 10 ) 

'^k= Yl '^iRi,k (vr), k> 1, 

, i=0 
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where xq (vt) is the stationary probability vector of the censored Markov chain 'ho (^) to 
level 0, and the scalar r is determined by YlT=o ~ ^ uniquely. 

Using the expression (|10l) of the fixed point vr, we set up an important relation as 
follows: 

TT = rxo (vr), VToi^o,! (vr), ^ TTiRi^k (vr), ^ 'KiRi,k (vr),... , (11) 

V i=0 i=0 ) 

which is called a fixed point equation with i?-measure. 

In what follows we consider two special cases in order to further explain the fixed point 
equation (1111) with i?-measure. 

Case one: Nonlinear Markov processes of GI/M/1 type 

In this case, the inhnitesimal generator F (vr) is given by 


/ 


r(vr) = 


Bi (vr) Bq (vr) 

B 2 (vr) Ai (vr) ^0 (vr) 

B 3 (vr) A 2 (vr) Ai (vr) ^0 (vr) 


\ 


V 


/ 


Let R (vr) be the minimal nonnegative solution to the nonlinear matrix equation 

00 

i^)Ak (vr) = 0. 

Then 


A:=0 


TTk = ttiR’^ ^ (vr), k>l, 

where the two vectors vtq and vri satisfy the following system of nonlinear matrix equations 


(vro,vri) 


Bi (vr) 


Bo (vr) 




CO CO 

D. _ TDk , 


x; R^ (vr) Bk +2 (vr) Xl R (^) Ak+i (vr) 
k=0 k=0 


= 0 


and 


vToe + TTi[I — R (vr)] ^ e = 1. 

Thus, the fixed point equation (jllj) with i?-measure is simplified as 

vr = (vro, vri, vrii? (vr), ttiR^ (vr),...) . 

Case two: Nonlinear Markov processes of M/G/1 type 
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In this case, the infinitesimal generator T (tt) is given by 



Bi (vr) 

B 2 (vr) 

B 3 (vr) 

B 4 (vr) 

... \ 


Bo (vr) 

Ai (vr) 

^2 (vr) 

^3 (vr) 


r(vr) = 


^0 (vr) 

Ai (vr) 

A 2 (vr) 





^0 (vr) 

Ai (vr) 



V 




/ 


Let G (vr) be the minimal nonnegative solution to the nonlinear matrix equation 


oo 

(vr) (vr) =0. 

fc=o 

Then 

OO 

^0 (vr) = (vr) + ^Bk (vr) (vr) Gi (vr), 
k=2 

oo 

'if (vr) = Ai (vr) + ^ (vr) (vr); 
k=2 

and the /^-measure 


for i > 1 


Bo,j (vr) = 


Rj (vr) = 


Sfc(vr)G"-i 

k=j+l 


vr 


[-'I' (tt)] \ i > 1, 


k=j+l 


[-^ (vr)] S j >1. 


The fixed point vr is given by 


vro = TXo (vr), 

< k—l 

vTfc = vro-Ro,fc (vr) + E T^iRk-i (vr), k>l, 

, i=l 

where xq (vr) is the stationary probability vector of the censored Markov chain Tq (vr) to 
level 0, and the scalar r is determined by YlT=o ~ ^ uniquely. Thus, the fixed point 
equation (llip with i?-measure is simplified as 


vr = rxo (vr), vroi?o,i (vr), vroi 2 o ,2 (vr) + vrii?i (vr), vroi?o,3 (vr) + ^ T^iRk-i (vr),... 

V i=i 

Now, we write the fixed point equation (llip with i?-measure as a functional form; 
vr = F (91 (vr)) where 91 (vr) is related to the measure, as shown in the above two special 
cases. Based on this, we can provide an approximative algorithm as follows: 
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Algorithm I: Computation of the fixed points 

Step one: Taking any initial probability vector: G (fl). 

Step two: Computing the infinitesimal generator: T and then compute the 

/^-measure, which gives = F (iH 

Step three: For N >2, compute = F (iH 

Step four: For a sufficiently small e > 0, if < e, then the computa¬ 

tion is over; otherwise we go to Step three. 

Note that it is possible for some practical big networks that there exist multiple fixed 
points because the infinitesimal generator F (tt) is more general, e.g., see such examples 
given in Marbukh m. Gibbens at al. [iD], Kelly m (see Page 349), Dobrushin [27] . 
Aiituiies et al. |3| and Tibi m- In this case, it is a key to design a suitable initial 
probability vector: ^ (fl), for example: 

(1) For any given integer m > 1, we take 


(°) = (11 1 0 


TT^ 

\m m m 

(2) For any given parameter p G (0,1), we take 


= {l- p,p{l-p),p‘^{l-p),p^{l-p),...) . 


(3) For any given parameter A > 0, we take 


TT 


( 0 ) 



-aA e-^1 

1 !’ 2 ! 



(4) It is obvious that we can construct a suitable vector through some practical 
observation and experience, for instance, a discrete-time PH distribution of order 2 is 
established by means of only the first three moments of a random variable. 

Now, we provide another algorithm for computing the fixed points. To do end, we set 
up a characteristic equation of the censoring matrix Tq (tt) to level 0, while the character¬ 
istic equation is satisfied by the fixed points. 

Note that for the censored Markov chain 4^0 (tt) to level 0, we have 


xo M ^0 (tt) = 0, xo (vr) e = 1; 


TTo = TXo (vr), TToe = r G (0,1). 

Thus it is easy to see from the irreducibility of the matrix F (vr) that the matrix Tq (vr) 
of size mo is also irreducible, hence this gives rank('ko (vr)) = mo — 1- For the M matrix 
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'I'o(7r), its eigenvalue with the maximal real part is equal to zero due to 'I'o(7r)e = 0. 
Let the characteristic equation be fx (vr) = det {xl — 'I'o (tt)) = 0. Then the fixed points 
satisfy the characteristic equation /o (tt) = det (—Tq (tt)) = 0, and hence det (Tq (vr)) = 0. 
Hence the fixed points satisfy the system of nonlinear equations as follows: 

f det (To (tt)) = 0, 

I rank (Tq (vr)) = mo — 1. 

Note that (1121) provide another algorithm for computing the fixed points as follows: 

Algorithm II: Computation of the fixed points 

Step one: Providing a numerical solution vf to the nonlinear characteristic equation: 
det (To (vr)) = 0. 

Step two: Check whether rank(To (vf)) = mo — 1. If Yes, then vf is a fixed point. If 
No, then going to Step one. 


4 Stability and Metastability 


In this section, we first discuss the Birkhoff center and the locally stable fixed points 
of the dynamic system of mean-field equations: ^pit) = p{t)T{p{t)) with p{0) = q. 
Then we apply the Lyapunov functions and the relative entropy to studying the stability 
or metastability of the big networks. Furthermore, we provide several interesting open 
problems with detailed interpretation. 

We write 

S = {vr : vrT (vr) =0, vre = 1} . 


Then it is clear that 


S = J vr : vr = txq (vr), vroi?o,i (^r), ^ (vr), ^ TTiRi^k (vr) 


vre = 1 


i=0 


i=0 


or 

§ = (vr : det (Tq (vr)) = 0, rank (Tq (vr)) = mo — 1, vre = 1} 

k-l 

with vTo = txq (vr) and vr^ = ^ '^iRi,k ('^) for A: > 1. 

i=0 

Since the vector equation vrT (vr) = 0, together with vre = 1, is nonlinear, it is possible 
for some practical big networks that there exist multiple elements in the set S. Such 
practical examples include Marbukh [67], Gibbens at al. [ID], Kelly [17] (see Page 349), 
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Dobrushin m. Antunes et al. [3] and Tibi [79]. At the same time, an argument by analytic 
functions can indicate an important result: The elements of the set S are isolated. 

To describe the isolated element structure of the set S, we often need to use the Birkhoff 
center of the dynamic system of mean-field equations. Notice that the Birkhoff center is 
used to check whether the fixed point is unique or not. Based on this, our discussion 
includes the following two cases: 

Case one: N —>■ oo. In this case, we denote by ‘h (t) a solution to the system of 
differential equations ^p{t) = p{t)T {p{t)) with p (0) = q. Thus, the Birkhoff center of 
the solution <I> (t) is dehned as 

© = < P G ^ (n) : P = lim <I> (t^) for any scale sequence 

k^oo 

{tk} with ti > 0 for I > 1 and lim tk = +oo 

/c—>-oo 

Notice that perhaps 0 contains the limit cycles or the equilibrium points (the local minimal 
points, or the local maximal points, or the saddle points). Thus it is clear that S C 0. 
Obviously, the limiting empirical Markov process {Y (t) : t > 0} spends most of its time 
in the Birkhoff center 0, where Y (t) = limjv^oo {t) weakly. 

Case two: t —>• +oo. In this case, we write 


^ .^gakly, 

t—>-+oo 

if for each N = 1, 2,3,..., the system of N weakly interacting big networks is stable. 
Let 

H = < ^ G fp (n) : If = lim for any positive integer sequence 

I k^oo 

{Nfc} with 1 < Ni < N 2 < N 3 < • • • and lim Nj. = 00 I . 

k^oo J 

It is easy to see that 


S C H C 0. 


At the same time, it is clear that 


S = {the local minimal points in ©} U {the local maximal points in 0} 
U {the saddle points in 0} . 


and 


0—§ = {the limit cycles in 0} . 


16 


In what follows, we discuss stability or metastability of the big networks. 

To analyze the stability or metastability, a key is to determine a Lyapunov function 
for the dynamic system of mean-field equations: ^p{t) = p{t)T{p{t)) with p{0) = q. 
The Lyapunov function g defined on (fl) is constructed such that 

yr{y)-Vg{y)<0, y € ^ (0). (13) 

It is easy to see that if tt G §, then ttF (tt) • Vg (vr) = 0 due to the fact that ttF (tt) = 0. 
On the other hand, if vrF (vr) • Vg (tt) = 0, then tt G §. 

Let |S| be the number of elements in the set §. If |§| = 1, then 

lim lim (t) = lim lim (t) = tt, weakly. 

N—¥oo t—y-\-oo N—^co 

If |§| > 2, then the system of big networks exhibits a metastability property, that 
is, the evolutionary process of the tagged big network with a mean-field modification 
switches from one stable point to the other one after a long residence time. In the study 
of metastability, it is a key to estimate the expected value of such a residence time. See 
Bovier |15] . and Olivieri and Vares m for more details. 

An interesting issue in the study of big networks is to analyze stability or metastability 
of the corresponding nonlinear Markov processes. On this line, it is a key to construct a 
Lyapunov function or a local Lyapunov function. Note that the relative entropy function 
in some sense can define a globally attracting Lyapunov function. 

For p,q (fl), we define the relative entropy of p with respect to q as 

R{p\\q) = '^Pxlog . 

x€Q 

(a) The linear case 

Let p {t) and q (t) be two different solutions to the ordinary differential equation 

^p{t)=p{t)A, p{0)=q, 

where A is the infinitesimal generator of an irreducible continuous-time Markov process. 
In this case, Dupuis and Fischer [30] indicated that 

iR(p(t)ll9(()) = - E 

x,y£Q, 
x^y 


Py jt) qx jt) 
Px (t) qy {t) 


^ qx (t) 


y,x 


< 0 , 


(14) 
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where ^ {z) = z log z — z + 1. At the same time, they indicated that {p (t) | |g (t)) = 0 
if and only if p{t) = q (t) for t > 0; and also -^R {p (t) | |7r) =0 if and only if p{t) = tt for 
t > 0. 

(b) The nonlinear case 

For the ordinary differential equation: = p{t)T {p{t)) with p{0) = q, Dupuis 

and Fischer m demonstrated that the the relative entropy relation (jl4p can not be applied 
directly. In this case, they hrst defined (t) as the state probability of the system of 
N big networks at time t > 0, and let (0) = 'S)^q- Based on this, they gave an 
approximate method to construct the Lyapunov function as follows: 

F{q)= lim lim ^R (0) \ {T)) = lim vr) . 

N^ooT^+oo N V / N^oo N ^ ’ 

For applying the relative entropy to construct a Lyapunov function, readers may also refer 

to Budhiraja et al. for more details. 

Now, we introduce the locally stable fixed points in the set S, and explain convergence 

of the sequence of Markov processes. 

If there exists only an element vr in the set S, then 

IJ,^^\t)^p{t), as A" —>■ oo, (15) 

and 

p (t) —>■ vr, as t —>■+ 00 . (16) 

When there exist multiple elements in the set S, we hope to find a similar property 
to that in (fTSl) and (fT6]l . To this end, it is necessary to introduce the locally stable fixed 
points, which are hrst dehned in Budhiraja et al. [18]. For convenience of readers, we 
restate the dehnition of locally stable hxed points here. 

We dehne the relative interior of of D as 

= {g E : g > 0} . 

Notice that the Markov chain F (vr) is irreducible, it is easy to see from the system of hxed 
point equations vrF (vr) = 0 and vre = 1 that any hxed point vr > 0, hence we obtain S C 

do. 

Definition 1 A fixed point vteD^ is said to be locally stable if there exists a relative open 
set ^ of that contains it and has the property that whenever p (0) = q€d, the solution 
p (t) to the dynamic system of mean-field equations (0) and converges to tt as t ^ +oo. 
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and 


Based on Definition (TJ every element in the set S is a locally stable fixed point. In this 
case, let be a relative open set of D for A: > 1, and we write 

S = ■ ■ 'I j G for A: > 1, 

= for A; > 1. 

as iV —)> oo, 

(t) => p (t, , as iV —>■ oo, 


If for each A: > 1 


then 


and 


(17) 

(18) 


P —>■ TT^^\ as A —>■ +oo, 

where p [t,q^^^'^ denotes that this solution p{t) depends on the initial vector q^^^ G 
where p (O, q^^^) = q^^K 

From the above analysis, for each locally stable fixed point in §, we find the similar 
property (HID and m to that in m and (USD. Based on this, computation of the 
locally stable fixed points is on a unified line, but the the initial vector q^^'^ G with 
^(Ar)(o) ^ q{k) 

as N ^ oo has a large impact on the limit of the solution p (A) as 
A +00, thus a basic task for computing the locally stable fixed points is to find a suitable 
collection of the relative open sets of D as follows: 


J' = 




which sufficiently corresponds to the set S = ...}. 

In the remainder of this section, we provide several interesting open problems with 
detailed interpretation. 

Open problem one: The mean drift condition. 

We consider an irreducible QBD process whose infinitesimal generator is given by 

/ Blip) Bo ip) ^ 

B2 ip) ip) ^0 ip) 

A2 ip) ip) Ao (p) 


r(p) = 


V 
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where T {p) e = 0, the sizes of the matrices Bi (p) and Ai (p) are niQ and m, respectively, 
and the sizes of other matrices can be determined accordingly. We assume that for any 
p € ip (n), the Markov process: A (p) = Aq (p) + Ai (p) + A 2 (p), is irreducible, aperiodic 
and positive recurrent. Let 6p be the stationary probability vector of the the Markov 
process A(p). Then it is clear that for for any p € ^(Q), the Markov process T (p) is 
positive recurrent if and only if 0pA2 (p) e > ffpAg (p) e. 

It is interesting to study how the mean drift condition: 0pA2 (p) e > ffpAg (p) e for any 
p G fp (n), can influence stability or metastability of the ordinary differential equation: 
^p (t) =p(t)r (p (t)) with p (0) = q. 

Open problem two: The censoring Markov processes. 

For the infinitesimal generator T (p) given in ([7|), it is easy to give the infinitesimal 
generator Tq (p) of the censoring Markov processes to level 0. It is very interesting (but 
difficult) to set up some useful relations of stability or metastability between two ordinary 
differential equations: ^p (t) = p(t) F (p (t)) and ^po (t) = po (t) Tq (p (t)), where p (t) = 
(Po (t),Pi (i),P2 (t),...). 

5 Concluding Remarks 

This paper sets up a broad class of nonlinear continuous-time block-structured Markov 
processes by means of applying the mean-field theory to the study of big networks, and 
proposes some effective algorithms for computing the fixed points of the nonlinear Markov 
process by means of the UL-type i2G-factorization. Furthermore, this paper considers 
stability or metastability of the big networks, and gives several interesting open problems 
with detailed interpretation. Along such a line, there are a number of interesting directions 
for potential future research, for example: 

• providing algorithms for computing the hxed points of big networks with multiple 
stable points; 

• studying the influence of the censoring Markov processes on the metastability; 

• discussing how to apply the i2G-factorizations given in Li [56] to compute the ex¬ 
pected residence times in the study of metastability; and 
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analyzing some big networks with a heterogeneous geographical environment, and 
set up their simultaneous systems of nonlinear Markov processes. 
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